Image capture and post-capture processing

ABSTRACT

Image data of a scene is captured. Spectral profile information is obtained for the scene. A database of plural spectral profiles is accessed, each of which maps a material to a corresponding spectral profile reflected therefrom. The spectral profile information for the scene is matched against the database, and materials for objects in the scene are identified by using matches between the spectral profile information for the scene against the database. Metadata which identifies materials for objects in the scene is constructed, and the metadata is embedded with the image data for the scene.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.13/033,578, filed Feb. 23, 2011, and U.S. application Ser. No.13/079,677, filed Apr. 4, 2011, and claims the benefit of suchapplications, the contents of which are hereby incorporated by referenceas if fully stated herein.

FIELD

The present disclosure relates to image capture and to post-captureprocessing such as rendering of the captured image.

BACKGROUND

In the field of image capture, it may be desirable to adjust photographsaccording to the material being photographed. For example, if a model iswearing a black velvet jacket over black leather pants, the photographermight want to render the final photo in such a manner as todifferentiate the jacket from the pants.

SUMMARY

Current photographic processes make little distinction between similarcolors, and the routine post-shooting edition typically cannot rely onlyon camera signals to identify areas of the image (e.g., making adistinction between a black velvet jacket over black leather pants).Accordingly, an artist or photographer must attempt to visually identifydistinct areas for separate post-processing, which can be difficult andtime-consuming.

The foregoing situation is addressed by matching spectral profiles ofobjects in a scene so as to identify materials for the objects, andstoring the identity of the materials in metadata together with imagedata for the scene for use during post-capture rendering.

Thus, in an example embodiment described herein, image data of a sceneis captured. Spectral profile information is obtained for the scene. Adatabase of plural spectral profiles is accessed, each of which maps amaterial to a corresponding spectral profile reflected therefrom. Thespectral profile information for the scene is matched against thedatabase, and materials for objects in the scene are identified by usingmatches between the spectral profile information for the scene againstthe database. Metadata which identifies materials for objects in thescene is constructed, and the metadata is embedded with the image datafor the scene.

By matching spectral profiles of objects in a scene so as to identifymaterials for the objects, and storing the materials in metadatatogether with image data for the scene for use during post-capturerendering, it is ordinarily possible to automatically identify distinctareas of an image for separate post-processing, without requiring theintervention of an artist or photographer.

According to another architecture proposed herein, during image capturea preview of a scene is displayed, and a user is provided with a userinterface to select regions in the scene for which to capture spectralprofile information. Spectral profiles of objects in the scene whichfall within the region of interest are matched so as to identifymaterials for the objects, and the materials are stored in metadatatogether with image data for the region of interest for use duringpost-capture rendering.

Thus, in another example embodiment described herein, preview image dataof a scene is captured. A designation of a region of interest isaccepted in the preview image data. Spectral image data of the scene iscaptured, and spectral profile information for the region of interest iscalculated by using the captured spectral image data for the scene. Adatabase of plural spectral profiles is accessed, of which each profilemaps a material to a corresponding spectral profile reflected therefrom.The spectral profile information for the region of interest is matchedagainst the database, and materials for objects in the region ofinterest are identified by using matches between the spectral profileinformation for the region of interest against the database. Metadatawhich identifies materials for objects in the region of interest andwhich identifies location of the region of interest relative to thescene is constructed. The metadata is stored together with image datafor the scene.

By matching spectral profiles of objects in a selected region ofinterest so as to identify materials for the objects, and storing thematerials in metadata together with image data for the selected regionof interest for use during post-capture rendering, it is ordinarilypossible to automatically identify distinct areas of an image forseparate post-processing without requiring the intervention of an artistor photographer. In addition, because the calculation and storage ofspectral profile information can be limited to the region of interest,it is ordinarily possible to conserve memory and processing resources.

This brief summary has been provided so that the nature of thisdisclosure may be understood quickly. A more complete understanding canbe obtained by reference to the following detailed description and tothe attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are views depicting an external appearance of an imagecapture device according to an example embodiment.

FIGS. 1C to 1G are views for explaining an imaging system according toexample embodiments.

FIGS. 2A to 2C are detailed block diagrams for explaining the internalarchitecture of the image capture device shown in FIG. 1 according toexample embodiments.

FIGS. 3A and 3B are views for explaining an image capture moduleaccording to example embodiments.

FIGS. 4A and 4B are flow diagrams for explaining processing in the imagecapture device shown in FIG. 1 according to example embodiments.

FIG. 5A is a view for explaining display of an image by the imagecapture device shown in FIG. 1 according to one example embodiment.

FIG. 5B is a view for explaining selection of a region of interestaccording to one example embodiment.

FIG. 5C is a view for explaining an output display for a selected regionof interest according to one example embodiment.

FIG. 6 is a view for explaining spectral reflectance factors accordingto an example embodiment.

FIG. 7 is a view for explaining a spectral power distribution accordingto one example embodiment.

FIG. 8 is a view for explaining spectral sensitivity curves according toan example embodiment.

FIG. 9 is a view for explaining a database of plural spectral profilesaccording to an example embodiment.

FIG. 10 is a view for explaining eigenvectors of the database of FIG. 9according to an example embodiment.

FIG. 11 is a view for explaining the use of spectral reflectances toidentify distinct areas in a captured image.

FIGS. 12 and 13 are views for explaining metadata according to exampleembodiments.

DETAILED DESCRIPTION

In the following example embodiments, there is described amulti-spectral digital camera which may be a digital still camera or adigital video camera. It is understood, however, that the followingdescription encompasses arbitrary arrangements which can incorporate orutilize imaging assemblies having a spectral response, for instance, adata processing apparatus having an image sensing function (e.g., apersonal computer) or a portable terminal having an image sensingfunction (e.g., a mobile telephone).

FIGS. 1A and 1B are views showing an example of an external appearanceof an image capture device 100 according to an example embodiment. Notein these figures, some components are omitted for conciseness. A useroperates buttons and switches 66, 310 to 312 and 351 to 357 for turningON/OFF the power of the digital camera 100, for setting, changing orconfirming the shooting parameters, for confirming the status of thecamera, and for confirming shot images.

Optical finder 104 is a viewfinder, through which a user can view ascene to be captured. In this embodiment optical finder 104 is separatefrom image display unit 28, but in some embodiments image display unit28 may also function as a viewfinder.

Flash (flash emission device) 48 is for emitting auxiliary light toilluminate a scene to be captured, if necessary.

Image sensor 14 that is inside camera 100 is an image sensor whichconverts an optical image into an electrical signal. In someembodiments, image sensor 14 may be tunable in accordance with a captureparameter. Image sensor 14 will be described more fully below withrespect to FIG. 2A.

Imaging system 150 is a camera system which is incorporated with theimage sensor 14 in order to provide additional capabilities forcapturing spectral information. In that regard, several arrangements arepossible for imaging system 150, including a monochrome imaging sensorcombined with a filter wheel or a liquid crystal tunable filter, anabsorption filter, an additional array of spectral sensing devices, or acolor imaging system with tunable spectral sensitivities. These exampleembodiments are described more fully below with respect to FIGS. 1C to1G. In addition, in another embodiment, image sensor 14 itself may beable to capture higher-resolution spectral data (e.g., higher than thethree channels for RGB). Imaging system 150 also could be an array ofhigh-spectral resolution sensors that directly measure spectralinformation such as based on metal waveguides producing surface plasmonpolaritons.

The power button 311 is provided to start or stop the digital camera100, or to turn ON/OFF the main power of the digital camera 100. Themenu button 352 is provided to display the setting menu such as shootingparameters and operation modes of the digital camera 100, and to displaythe status of the digital camera 100. The menu includes selectable itemsor items whose values are variable.

A delete button 351 is pressed for deleting an image displayed on aplayback mode or a shot-image confirmation screen. In the presentembodiment, the shot-image confirmation screen (a so-called quick reviewscreen) is provided to display a shot image on the image display unit 28immediately after shooting for confirming the shot result. Furthermore,the present embodiment is constructed in a way that the shot-imageconfirmation screen is displayed as long as a user keeps pressing theshutter button 310 after the user instructs shooting by shutter buttondepression.

An enter button 353 is pressed for selecting a mode or an item. When theenter button 353 is pressed, the system controller 50 in FIG. 2A setsthe mode or item selected at this time. The display ON/OFF button 66 isused for selecting displaying or non-displaying of photographinformation regarding the shot image, and for switching the imagedisplay unit 28 to be functioned as an electronic view finder.

A left button 354, a right button 355, an up button 356, and a downbutton 357 may be used for the following purposes, for instance,changing an option (e.g., items, images) selected from plural options,changing an index position that specifies a selected option, andincreasing or decreasing numeric values (e.g., correction value, dateand time).

Half-stroke of the shutter button 310 instructs the system controller 50to start, for instance, AF processing, AE processing, AWB processing, EFprocessing or the like. Full-stroke of the shutter button 310 instructsthe system controller 50 to perform shooting.

The zoom operation unit 65 is operated by a user for changing the angleof view (zooming magnification or shooting magnification).

A recording/playback selection switch 312 is used for switching arecording mode to a playback mode, or switching a playback mode to arecording mode. Note, in place of the above-described operation system,a dial switch may be adopted or other operation systems may be adopted.

FIGS. 1C to 1G are views for explaining an imaging system (e.g., imagingsystem 150) for capturing spectral information according to exampleembodiments. These embodiments are shown merely for purposes of example,and other arrangements are possible. In that regard, as mentioned above,in some embodiments image sensor 14 may be constructed to capturehigh-resolution additional spectral data itself, and thus in some casesthe additional hardware of imaging system 150 may not be necessary.

FIGS. 1C and 1D depict embodiments in which image sensor 14 is an RGBsensor combined with an additional imaging sensor. The additionalimaging sensor is comprised of a monochrome sensor 151 and a set ofnarrow-band filters. The narrow-band filters, in turn, can be comprisedof a filter wheel 152 (FIG. 1C) with filters with different spectralbands, or a liquid crystal tunable filter 153 (FIG. 1D). Either of theseembodiments ordinarily provides relatively high spectral resolution andrelatively high spatial resolution. However, due to cost and size of thesystem, such embodiments ordinarily are only appropriate for high-endimaging of static objects.

FIG. 1E depicts an embodiment in which image sensor 14 is an RGB sensorcombined with an absorption filter 154, for example as shown in U.S.Pat. No. 7,554,586, “System and method for scene image acquisition andspectral estimation using a wide-band multi-channel image capture”, thecontents of which are incorporated by reference herein. The captured RGBfrom image sensor 14 without an external filter provides the traditionalimage capture. Meanwhile, a spectral reflectance estimation process isperformed to get higher spectral resolution data from lower spectralresolution captured data provided by the combination of unfilteredimages from image sensor 14, and filtered RGB images from absorptionfilter 154. The external absorption filter 154 changes the overallsensitivities of the original RGB sensor providing three additionalchannels. This embodiment provides relatively high spatial resolutionand is relatively usable for dynamic scenes if the filter 154 isfast-switching, and there is ordinarily no need for a secondary sensoras in the embodiments of FIGS. 1C and 1D. On the other hand, theembodiment of FIG. 1E tends to have relatively low spectral resolution.

FIG. 1F depicts an embodiment in which image sensor 14 is an RGB sensorcombined with an additional high-spectral resolution but low-spatialresolution imaging device 156, for example a device which includes anarray of spectral sensing devices 155 with high-spectral resolution,such as described in U.S. Publications No. 2010/0045050, 2010/0046077,2010/0053755 and 2010/0182598, the contents of which are incorporated byreference herein. Main RGB imaging sensor 14 provides the conventionalphotography capture, whereas a secondary sensor (array of high-spectralresolution sensors) 155 works as a low-spatial resolution buthigh-spectral resolution spectral measurement device. The arrangement ofFIG. 1F provides high spectral resolution with relatively low cost, andcan be applied to dynamic scenes. On the other hand, the secondarysensor (e.g., the array of spectral sensing devices) ordinarily has alow spatial resolution.

FIG. 1G depicts an example embodiment in which image sensor 14 is an RGBimaging sensor coupled with a color imaging system 157 with tunablespectral sensitivities. The tunable spectral sensitivities may betunable in accordance with a capture parameter 17. This arrangement isdescribed in detail in U.S. application Ser. No. 12/949,592, byFrancisco Imai, entitled “Adaptive Spectral Imaging By Using An ImagingAssembly With Tunable Spectral Sensitivities”, the contents of which areincorporated by reference herein.

As mentioned above, image sensor 14 itself may have high spectralresolution and capture additional multi-spectral data. Thus, additionalhardware might not be necessary at all, although multiple captures mightbe needed. Regardless of the implementation, the spatial resolution ofthe captured image will be higher than the spectral resolution of thecaptured image.

Additionally, image sensor 14 itself could have tunable spectralsensitivities, as described in U.S. application Ser. No. 12/949,592. Insuch an embodiment, image sensor 14 is a multi-spectral image sensorwhich has a spectral response which is tunable in accordance with acapture parameter 17.

In that regard, any of the embodiments above ordinarily will provideenough spectral information to identify, or at least differentiatebetween, different materials in a scene. As mentioned, some embodimentsmay capture lower resolution spectral resolution than others, and thushave less accuracy in identifying materials. Nevertheless, even lowspectral resolution information may allow for differentiation betweendistinct areas comprised of different materials.

FIG. 2A is a block diagram showing an example of the arrangement of themulti-spectral digital camera 100 as an image capture device accordingto this embodiment. Referring to FIG. 2, reference numeral 10 denotes animaging lens; 12, a shutter having an aperture function; and 14, animage sensor which converts an optical image into an electrical signal.Reference numeral 16 denotes an A/D converter which converts an analogsignal into a digital signal. The A/D converter 16 is used when ananalog signal output from the image sensor 14 is converted into adigital signal and when an analog signal output from an audio controller11 is converted into a digital signal. Reference numeral 102 denotes ashield, or barrier, which covers the image sensor including the lens 10of the digital camera 100 to prevent an image capturing system includingthe lens 10, shutter 12, and image sensor 14 from being contaminated ordamaged.

In FIG. 2A, an imaging assembly is comprised of image sensor 14 andassociated optics, such that in some embodiments the imaging assembly iscomprised of image sensor 14 and lens 10.

The optical system 10 may be of a zoom lens, thereby providing anoptical zoom function. The optical zoom function is realized by drivinga magnification-variable lens of the optical system 10 using a drivingmechanism of the optical system 10 or a driving mechanism provided onthe main unit of the digital camera 100.

A light beam (light beam incident upon the angle of view of the lens)from an object in a scene that goes through the optical system (imagesensing lens) 10 passes through an opening of a shutter 12 having adiaphragm function, and forms an optical image of the object on theimage sensing surface of the image sensor 14. The image sensor 14converts the optical image to analog image signals and outputs thesignals to an A/D converter 16. The A/D converter 16 converts the analogimage signals to digital image signals (image data). The image sensor 14and the A/D converter 16 are controlled by clock signals and controlsignals provided by a timing generator 18. The timing generator 18 iscontrolled by a memory controller 22 and a system controller 50.

In the embodiment shown in FIG. 2A, image sensor 14 is tunable inaccordance with a capture parameter 17. The precise nature of thespectral responsivity of image sensor 14 is controlled via captureparameter 17. In this embodiment, capture parameter 17 may be comprisedof multiple spatial masks, with one mask each for each channel ofinformation output by image sensor 14. Each spatial mask comprises anarray of control parameters corresponding to pixels or regions of pixelsin image sensor 14. In this regard, image sensor 14 may be comprised ofa transverse field detector (TFD) sensor mentioned hereinabove. Thespatial masks may correspond to voltage biases applied to controlelectrodes of the TFD sensor. The spectral responsivity of each pixel,or each region of plural pixels, is thus tunable individually andindependently of other pixels or regions of pixels.

In one example embodiment, image sensor 14 can gather high-resolutionspectral data, and outputs, for example, five or more channels of colorinformation, including a red-like channel, a green-yellow-like channel,a green-like channel, a blue-green-like channel, and a blue-likechannel. In such an example, where image sensor 14 outputs five or morechannels, capture parameter 17 includes a spatial mask DR for thered-like channel of information, a spatial mask DGY for thegreen-yellow-like channel of information, a spatial mask DG for thegreen-like channel of information, a spatial mask DBG for theblue-green-like channel of information and a spatial mask DB for theblue-like channel of information.

In the embodiment shown in FIG. 2A, however, it can be assumed thatimage sensor 14 is a conventional RGB sensor which is combined withimaging system 150 in FIG. 2 to gather the additional spectralinformation.

Imaging system 150 is a camera system which is incorporated with theimage sensor 14 in order to provide additional capabilities forcapturing spectral information. In that regard, several arrangements arepossible for imaging system 150, including a monochrome imaging sensorcombined with a filter wheel or a liquid crystal tunable filter, anabsorption filter, an additional array of spectral sensing devices, or acolor imaging system with tunable spectral sensitivities, as describedabove with respect to FIGS. 1C to 1G.

Reference numeral 18 denotes a timing generator, which supplies clocksignals and control signals to the image sensor 14, the audio controller11, the A/D converter 16, and a D/A converter 26. The timing generator18 is controlled by a memory controller 22 and system controller 50.Reference numeral 20 denotes an image processor, which applies resizeprocessing such as predetermined interpolation and reduction, and colorconversion processing to data from the A/D converter 16 or that from thememory controller 22. The image processor 20 executes predeterminedarithmetic processing using the captured image data, and the systemcontroller 50 executes exposure control and ranging control based on theobtained arithmetic result.

As a result, TTL (through-the-lens) AF (auto focus) processing, AE (autoexposure) processing, and EF (flash pre-emission) processing areexecuted. The image processor 20 further executes predeterminedarithmetic processing using the captured image data, and also executesTTL AWB (auto white balance) processing based on the obtained arithmeticresult. It is understood that in other embodiments, optical finder 104may be used in combination with the TTL arrangement, or in substitutiontherefor.

Output data from the A/D converter 16 is written in a memory 30 via theimage processor 20 and memory controller 22 or directly via the memorycontroller 22. The memory 30 stores image data which is captured by theimage sensor 14 and is converted into digital data by the A/D converter16, and image data to be displayed on an image display unit 28. Theimage display unit 28 may be a liquid crystal screen. Note that thememory 30 is also used to store audio data recorded via a microphone 13,still images, movies, and file headers upon forming image files.Therefore, the memory 30 has a storage capacity large enough to store apredetermined number of still image data, and movie data and audio datafor a predetermined period of time.

A compression/decompression unit 32 compresses or decompresses imagedata by adaptive discrete cosine transform (ADCT) or the like. Thecompression/decompression unit 32 loads captured image data stored inthe memory 30 in response to pressing of the shutter 310 as a trigger,executes the compression processing, and writes the processed data inthe memory 30. Also, the compression/decompression unit 32 appliesdecompression processing to compressed image data loaded from adetachable recording unit 202 or 212, as described below, and writes theprocessed data in the memory 30. Likewise, image data written in thememory 30 by the compression/decompression unit 32 is converted into afile by the system controller 50, and that file is recorded innonvolatile memory 56 and/or the recording unit 202 or 212, as alsodescribed below.

The memory 30 also serves as an image display memory (video memory).Reference numeral 26 denotes a D/A converter, which converts imagedisplay data stored in the memory 30 into an analog signal, and suppliesthat analog signal to the image display unit 28. Reference numeral 28denotes an image display unit, which makes display according to theanalog signal from the D/A converter 26 on the liquid crystal screen 28of an LCD display. In this manner, image data to be displayed written inthe memory 30 is displayed by the image display unit 28 via the D/Aconverter 26.

The exposure controller 40 controls the shutter 12 having a diaphragmfunction based on the data supplied from the system controller 50. Theexposure controller 40 may also have a flash exposure compensationfunction by linking up with flash (flash emission device) 48. The flash48 has an AF auxiliary light projection function and a flash exposurecompensation function.

The distance measurement controller 42 controls a focusing lens of theoptical system 10 based on the data supplied from the system controller50. A zoom controller 44 controls zooming of the optical system 10. Ashield controller 46 controls the operation of a shield (barrier) 102 toprotect the optical system 10.

Reference numeral 13 denotes a microphone. An audio signal output fromthe microphone 13 is supplied to the A/D converter 16 via the audiocontroller 11 which includes an amplifier and the like, is convertedinto a digital signal by the A/D converter 16, and is then stored in thememory 30 by the memory controller 22. On the other hand, audio data isloaded from the memory 30, and is converted into an analog signal by theD/A converter 26. The audio controller 11 drives a speaker 15 accordingto this analog signal, thus outputting a sound.

A nonvolatile memory 56 is an electrically erasable and recordablememory, and uses, for example, an EEPROM. The nonvolatile memory 56stores constants, computer-executable programs, and the like foroperation of system controller 50. Note that the programs include thosefor execution of various flowcharts.

In particular, as shown in FIG. 2B, non-volatile memory 56 is an exampleof a non-transitory computer-readable memory medium, having retrievablystored thereon image capture module 300 as described herein. Accordingto this example embodiment, the image capture module 300 includes atleast a capture module 301 for capturing image data of a scene, aobtaining module 302 for obtaining spectral profile information for thescene, an access module 303 for accessing a database of plural spectralprofiles each of which maps a material to a corresponding spectralprofile reflected therefrom, a matching module 304 for matching thespectral profile information for the scene against the database, anidentification module 305 for identifying materials for objects in thescene by using matches between the spectral profile information for thescene against the database, a construction module 306 for constructingmetadata which identifies materials for objects in the scene, and anembedding module 307 for embedding the metadata with the image data forthe scene. These modules will be discussed in more detail below withrespect to FIG. 3A.

Additionally, as shown in FIG. 2B, non-volatile memory 56 also includesimage data 251, which includes image data from a scene. The image datafor the scene may also be embedded with metadata which identifiesmaterials for objects in the scene. Non-volatile memory 56 furtherstores spectral profile information 252. Spectral profile information252 includes information indicating the spectral signature of objects inthe region of interest, and the respective profile information ismatched against a database of predetermined spectral profiles 253 inorder to identify the materials of the object. Each of these elementswill be described more fully below.

Reference numeral 50 denotes a system controller, which controls theentire digital camera 100. The system controller 50 executes programsrecorded in the aforementioned nonvolatile memory 56 to implementrespective processes to be described later of this embodiment. Referencenumeral 52 denotes a system memory which comprises a RAM. On the systemmemory 52, constants and variables required to operate system controller50, programs read out from the nonvolatile memory 56, and the like aremapped.

A mode selection switch 60, shutter switch 310, and operation unit 70form operation means used to input various operation instructions to thesystem controller 50.

The mode selection switch 60 includes the imaging/playback selectionswitch, and is used to switch the operation mode of the systemcontroller 50 to one of a still image recording mode, movie recordingmode, playback mode, and the like.

The shutter switch 62 is turned on in the middle of operation (halfstroke) of the shutter button 310 arranged on the digital camera 100,and generates a first shutter switch signal SW1. Also, the shutterswitch 64 is turned on upon completion of operation (full stroke) of theshutter button 310, and generates a second shutter switch signal SW2.The system controller 50 starts the operations of the AF (auto focus)processing, AE (auto exposure) processing, AWB (auto white balance)processing, EF (flash pre-emission) processing, and the like in responseto the first shutter switch signal SW1. Also, in response to the secondshutter switch signal SW2, the system controller 50 starts a series ofprocessing (shooting) including the following: processing to read imagesignals from the image sensor 14, convert the image signals into imagedata by the A/D converter 16, process the image data by the imageprocessor 20, and write the data in the memory 30 through the memorycontroller 22; and processing to read the image data from the memory 30,compress the image data by the compression/decompression circuit 32, andwrite the compressed image data in non-volatile memory 56, and/or inrecording medium 200 or 210.

A zoom operation unit 65 is an operation unit operated by a user forchanging the angle of view (zooming magnification or shootingmagnification). The operation unit 65 can be configured with, e.g., aslide-type or lever-type operation member, and a switch or a sensor fordetecting the operation of the member.

The image display ON/OFF switch 66 sets ON/OFF of the image display unit28. In shooting an image with the optical finder 104, the display of theimage display unit 28 configured with a TFT, an LCD or the like may beturned off to cut the power supply for the purpose of power saving.

The flash setting button 68 sets and changes the flash operation mode.In this embodiment, the settable modes include: auto, flash-on, red-eyereduction auto, and flash-on (red-eye reduction). In the auto mode,flash is automatically emitted in accordance with the lightness of anobject. In the flash-on mode, flash is always emitted whenever shootingis performed. In the red-eye reduction auto mode, flash is automaticallyemitted in accordance with lightness of an object, and in case of flashemission the red-eye reduction lamp is always emitted whenever shootingis performed. In the flash-on (red-eye reduction) mode, the red-eyereduction lamp and flash are always emitted.

The operation unit 70 comprises various buttons, touch panels and so on.More specifically, the operation unit 70 includes a menu button, a setbutton, a macro selection button, a multi-image reproduction/repagingbutton, a single-shot/serial shot/self-timer selection button, a forward(+) menu selection button, a backward (−) menu selection button, and thelike. Furthermore, the operation unit 70 may include a forward (+)reproduction image search button, a backward (−) reproduction imagesearch button, an image shooting quality selection button, an exposurecompensation button, a date/time set button, a compression mode switchand the like.

The compression mode switch is provided for setting or selecting acompression rate in JPEG (Joint Photographic Expert Group) compression,recording in a RAW mode and the like. In the RAW mode, analog imagesignals outputted by the image sensing device are digitalized (RAW data)as it is and recorded.

Note in the present embodiment, RAW data includes not only the dataobtained by performing A/D conversion on the photoelectrically converteddata from the image sensing device, but also the data obtained byperforming lossless compression on A/D converted data. Moreover, RAWdata indicates data maintaining output information from the imagesensing device without a loss. For instance, RAW data is A/D convertedanalog image signals which have not been subjected to white balanceprocessing, color separation processing for separating luminance signalsfrom color signals, or color interpolation processing. Furthermore, RAWdata is not limited to digitalized data, but may be of analog imagesignals obtained from the image sensing device.

According to the present embodiment, the JPEG compression mode includes,e.g., a normal mode and a fine mode. A user of the digital camera 100can select the normal mode in a case of placing a high value on the datasize of a shot image, and can select the fine mode in a case of placinga high value on the quality of a shot image.

In the JPEG compression mode, the compression/decompression circuit 32reads image data written in the memory 30 to perform compression at aset compression rate, and records the compressed data in, e.g., therecording medium 200.

In the RAW mode, analog image signals are read in units of line inaccordance with the pixel arrangement of the color filter of the imagesensor 14, and image data written in the memory 30 through the A/Dconverter 16 and the memory controller 22 is recorded in non-volatilememory 56, and/or in recording medium 200 or 210.

The digital camera 100 according to the present embodiment has aplural-image shooting mode, where plural image data can be recorded inresponse to a single shooting instruction by a user. Image datarecording in this mode includes image data recording typified by an autobracket mode, where shooting parameters such as white balance andexposure are changed step by step. It also includes recording of imagedata having different post-shooting image processing contents, forinstance, recording of plural image data having different data formssuch as recording in a JPEG form or a RAW form, recording of image datahaving the same form but different compression rates, and recording ofimage data on which predetermined image processing has been performedand has not been performed.

A power controller 80 comprises a power detection circuit, a DC-DCconverter, a switch circuit to select the block to be energized, and thelike. The power controller 80 detects the existence/absence of a powersource, the type of the power source, and a remaining battery powerlevel, controls the DC-DC converter based on the results of detectionand an instruction from the system controller 50, and supplies anecessary voltage to the respective blocks for a necessary period. Apower source 86 is a primary battery such as an alkaline battery or alithium battery, a secondary battery such as an NiCd battery, an NiMHbattery or an Li battery, an AC adapter, or the like. The main unit ofthe digital camera 100 and the power source 86 are connected byconnectors 82 and 84 respectively comprised therein.

The recording media 200 and 210 comprise: recording units 202 and 212that are configured with semiconductor memories, magnetic disks and thelike, interfaces 203 and 213 for communication with the digital camera100, and connectors 206 and 216. The recording media 200 and 210 areconnected to the digital camera 100 through connectors 206 and 216 ofthe media and connectors 92 and 96 of the digital camera 100. To theconnectors 92 and 96, interfaces 90 and 94 are connected. Theattached/detached state of the recording media 200 and 210 is detectedby a recording medium attached/detached state detector 98.

Note that although the digital camera 100 according to the presentembodiment comprises two systems of interfaces and connectors forconnecting the recording media, a single or plural arbitrary numbers ofinterfaces and connectors may be provided for connecting a recordingmedium. Further, interfaces and connectors pursuant to differentstandards may be provided for each system.

For the interfaces 90 and 94 as well as the connectors 92 and 96, cardsin conformity with a standard, e.g., PCMCIA cards, compact flash (CF)(registered trademark) cards and the like, may be used. In this case,connection utilizing various communication cards can realize mutualtransfer/reception of image data and control data attached to the imagedata between the digital camera and other peripheral devices such ascomputers and printers. The communication cards include, for instance, aLAN card, a modem card, a USB card, an IEEE 1394 card, a P1284 card, anSCSI card, and a communication card for PHS or the like.

The optical finder 104 is configured with, e.g., a TTL finder, whichforms an image from the light beam that has gone through the lens 10utilizing prisms and mirrors. By utilizing the optical finder 104, it ispossible to shoot an image without utilizing an electronic view finderfunction of the image display unit 28. The optical finder 104 includesindicators, which constitute part of image display unit 28, forindicating, e.g., a focus state, a camera shake warning, a flash chargestate, a shutter speed, an f-stop value, and exposure compensation.

A communication circuit 110 provides various communication functionssuch as USB, IEEE 1394, P1284, SCSI, modem, LAN, RS232C, and wirelesscommunication. To the communication circuit 110, a connector 112 can beconnected for connecting the digital camera 100 to other devices, or anantenna can be provided for wireless communication.

A real-time clock (RTC, not shown) may be provided to measure date andtime. The RTC holds an internal power supply unit independently of thepower supply controller 80, and continues time measurement even when thepower supply unit 86 is OFF. The system controller 50 sets a systemtimer using a date and time obtained from the RTC at the time ofactivation, and executes timer control.

FIG. 3A is a view for explaining an image capture module according toone example embodiment. As previously discussed with respect to FIG. 2B,image capture module 300 comprises computer-executable process stepsstored on a non-transitory computer-readable storage medium, such asnon-volatile memory 56. More or less modules may be used, and otherarchitectures are possible.

As shown in FIG. 3A, image capture module 300 at least a capture module301 for capturing image data of a scene. To that end, capture module 301communicates with image sensor 14 and/or imaging system 150, whichgathers image data and associated spectral information from a scene.Capture module 301 transmits the image data for the scene and thespectral information to obtaining module 302, for obtaining spectralprofile information for the scene (e.g., from image sensor 14 if imagesensor 14 can capture such data, or from imaging system 150 if imagesensor 14 is a conventional RGB sensor). Access module 303 accesses adatabase of plural spectral profiles, each of which maps a material to acorresponding spectral profile reflected therefrom. In that regard, thedatabase of plural spectral profiles may be stored in non-volatilememory 56, shown in FIG. 2B as database of spectral profiles 253.

Matching module 304 matches the spectral profile information for thescene calculated by obtaining module 302 against the database ofspectral profiles (e.g., database of spectral profiles 253), andtransmits this information to identification module 305. Identificationmodule 305 identifies materials for objects in the scene by usingmatches between the spectral profile information for the scene againstthe database. Once the materials corresponding to the spectral profileinformation are identified, construction module 306 constructs metadata(e.g., object metadata) which identifies materials for objects in thescene. Embedding module 307 embeds the metadata with the image data forthe scene. The resultant embedded image data may be stored with otherimage data, for example as image data 251 in non-volatile memory 56shown in FIG. 2B.

FIG. 4A is a flow diagram for explaining processing in the image capturedevice shown in FIG. 1 according to an example embodiment.

Briefly, in FIG. 4A, image data of a scene is captured. Spectral profileinformation is obtained for the scene. A database of plural spectralprofiles is accessed, each of which maps a material to a correspondingspectral profile reflected therefrom. The spectral profile informationfor the scene is matched against the database, and materials for objectsin the scene are identified by using matches between the spectralprofile information for the scene against the database. Metadata whichidentifies materials for objects in the scene is constructed, and themetadata is embedded with the image data for the scene.

In particular, in step 401, a user instructs image capture, for exampleby full-stroke of the shutter button 310.

In step 402, the image capture device captures image data. Inparticular, upon instruction of image capture, light beam (light beamincident upon the angle of view of the lens) from an object in a scenethat goes through the optical system (image sensing lens) 10 passesthrough an opening of the shutter 12 having a diaphragm function, andforms an optical image of the object on the image sensing surface of theimage sensor 14. The image sensor 14 converts the optical image toanalog image signals and outputs the signals to an A/D converter 16. TheA/D converter 16 converts the analog image signals to digital imagesignals (image data).

In addition, spectral information is captured along with the raw imagedata, by image sensor 14 (if image sensor 14 is capable of capturingsufficient spectral data on its own, the spectral profile informationfor the scene is calculated from the captured image data of the scene)or by a combination of image sensor 14 and imaging system 150 (if imagesensor 14 is not capable of capturing sufficient spectral data on itsown). Example embodiments for capturing the spectral information aredescribed above with respect to FIGS. 1C to 1G.

The spectral information may include, for example, five or more channelsof color information, including a red-like channel, a green-yellow-likechannel, a green-like channel, a blue-green-like channel, and ablue-like channel. The image data may be comprised of tri-stimulusdevice independent image data, e.g., XYZ image data.

In step 403, spectral profile information is obtained for the scene. Thespectral profile information may be obtained from spectral data fromimage sensor 14 (if capable of capturing sufficient spectral data on itsown) or a combination of image sensor 14 and imaging system 150 (ifimage sensor 14 is not capable of capturing sufficient spectral data onits own). For example, in an example embodiment in which each pixel hasfive channels, each pixel is integrated to produce five digital signals,one signal for each channel. Each channel is tuned to a spectral bandwithin the visible spectrum. Therefore, the digital signal for eachchannel corresponds to a respective spectral reflectance curve withinthe visible spectrum.

Thus, spectral data gathered by imaging system 150 (or image sensor 14,if acting alone) is converted into a spectral reflectance curve,generally in the range from 400 to 700 nm of visible light. In thatregard, spectral data may have up to 61 (with sampling rate of 5 nm) ormore separate values. Comparing all of these values can be relativelyinefficient. Accordingly, since spectral reflectance curves aregenerally smooth, it is ordinarily possible to use less values (i.e.,less than the 61 discrete values), and eigenvectors can be used toreduce the required processing.

By assuming the relative smoothness of most of spectral reflectancecurves it is possible to reduce the number of components of spectraldata to six eigenvectors by performing eigenvector analysis. Atransformation from the six capture signals to the coefficients ofeigenvectors can be produced by a training set of captured images ofobjects with known representative spectral reflectances. Once the imageis captured, the transformation is used to calculate the coefficients ofthe eigenvectors for each pixel of the image.

Specifically, eigenvectors and their coefficients represent the spectraldata. The pre-calculated eigenvectors are used to decompose the capturedspectral curves into coefficients, which can then be compared withcoefficients in the database. The pre-calculated eigenvectors can begenerated before image capture from common captured spectralreflectances, such as skin, clothes, hair and the like. Alternatively,eigenvectors could be pre-calculated for every possible reflectance,although this approach might require significant resources.

In one approach, the spectral reflectance of a collection of objectsR_(λ) _(—) _(collection) is statistically analyzed. Eigenvector analysisis performed and 6 eigenvectors e_(i) (where i=1 to 6) arepre-calculated. Any reflectance R_(λ) _(—) _(j) (where j=1 to m, where mis the number of objects in the collection) in the collection of objectscould be reconstructed by combining the eigenvectors e_(j).

Meanwhile, the estimation of the spectral reflectance for a capturedobject j is given by R_(λ) _(—) _(j) _(—) _(estimation)=Σa_(i)*e_(i)where a_(i) are the coefficients of the eigenvectors for object j. Thecoefficients of the eigenvectors (represented here by a vector A_(j)whose dimensions are i by 1) can be estimated from captured digitalsignals D_(j) of object j by a pre-calculated transformation T fromcaptured digital signals to eigenvectors: Aj=T*D_(j). Accordingly, it ispossible to obtain the coefficients of the eigenvectors from thecaptured spectral reflectance curves, which can then be compared withcoefficients of eigenvectors from the database of plural spectralprofiles to see if there is a match.

In some example embodiments such as that shown in FIG. 1F, due to thehigh number of components (e.g., R, G, B, and others) of the spectralinformation, it is difficult to deal with spectral data as signaturesfor objects. One possibility to deal with the burden of the high numberof components is by relating coefficients of eigenvectors Aj associatedto a particular object j.

In such a configuration, the measured spectra can be decomposed by thepre-calculated eigenvectors e_(i) as follows: Aj=R_(λ) _(—)_(j)*pinv(e_(i)), where pinv is the pseudo-inverse operation.

A concrete example of calculating spectral profile information from thecaptured image data for the scene will now be briefly described withrespect to FIGS. 6 to 11.

In this example, assume a model of African origin whose face skin has aspectral reflectance R_skin and who has black hair with spectralreflectance R_hair. The typical spectral reflectance curves are shown inFIG. 6. It is clear from FIG. 6 that hair and skin have very distinctspectral reflectance properties.

First, assume the model is imaged under typical photographic studiohalogen lamps (whose spectral power distribution is shown in FIG. 7) andthe model pictures are taken by a conventional professional digital SLRwhose typical red-green-blue spectral sensitivities are shown in FIG. 8.When the digital images are captured, they include average values ofRed_hair=24, Green_hair=14 and Blue_hair=7 for hair and average valuesof Red_skin=24, Green_skin=11 and Blue_skin=5 for skin. The cameravalues for dark skin and black hair are extremely similar, making themsomewhat undistinguishable.

On the other hand, an imaging system that has a secondary spectralmeasurement sensor (e.g., any of FIGS. 1C to 1G) or an image sensor 14with high spectral resolution captures spectral reflectance values formultiple regions of the image including hair and skin, respectivelyR_hair and R_skin. These measurements correspond to what is depicted inFIG. 6.

When the coefficient of eigenvectors are calculated for the capturedblack hair data it gives the following values are produced:A_hair=[0.006, −0.011, −0.001, −0.007, 0.017, 0.118], while the valuesfor dark skin are given by A_skin=[0.0002, −0.029, −0.027, −0.035,−0.043, 0.429]. In this case, the spectral signatures given by thecoefficients of eigenvectors are distinct between dark skin and blackhair. These eigenvectors are compared with a database of plural spectralprofiles such as database of spectral profiles 253 to identify materialsfor objects in the scene, as described more fully below with respect tosteps 404 to 406.

Returning to FIG. 4A, in step 404, a database of plural spectralprofiles is accessed. The database of plural spectral profiles may bestored in non-volatile memory 56, as shown by database of spectralprofiles 253 in FIG. 2B. In another embodiment, the database of pluralspectral profiles could be stored remotely in a server, provided thatsuch server can be accessed from image capture device 100, i.e., as longas image capture device 100 has remote data access capabilities. Each ofthe plural spectral profiles maps a material to a corresponding spectralprofile reflected therefrom.

FIG. 9 depicts an example of such a database. More specifically, FIG. 9depicts a spectral database (such as the Vrhel database: Vrhel, M. J.,R. Gershon, and L. S. Iwan, Measurement and analysis of objectreflectance spectra, Color Res. and Appl., 19, 4-9, 1994, the contentsof which are incorporated by reference herein. This database iscomprised by spectral measurement of 170 objects. In that regard, forpurposes of conciseness, the full database is not shown in FIG. 9. Thedatabase is one example of a pre-loaded set of spectral profiles in theform of computed eigenvectors and a look-up table (LUT) with typicalspectral signatures (coefficients of eigenvectors) of most commonlyimaged objects, such as skin, hair, vegetation, sky, etc.

Eigenvector analysis is performed for this collection of spectralreflectances, and the first 5 eigenvectors are shown in FIG. 10.

In step 405, the spectral profile information for the scene is matchedagainst the database.

In particular, the coefficients of eigenvectors calculated in step 403for the captured black hair data A_hair=[0.006, −0.011, −0.001, −0.007,0.017, 0.118] and the dark skin data A_skin=[0.0002, −0.029, −0.027,−0.035, −0.043, 0.429] are compared with the plural profiles of spectralsignatures accessed in step 404 to see if there are matches withspectral signatures of pre-identified objects in the database. If thereare matches, the respective spectral signatures are then used to segmentareas of the image with different spectral properties.

In that regard, the spectral profiles may be comprised of a relativelylow number of spectral components. In particular, it may be unnecessaryand impractical to attempt to specifically identify the exact materialfor each object in the scene. For example, outside of a specific settingin which all potential materials are known, it may not be possible tospecifically identify an exact material, as this would require anenormous database of plural spectral profiles for all possiblematerials.

Nevertheless, even spectral profiles comprised of a relatively lownumber of spectral components can be used to differentiate betweendistinct areas made up of different materials, so that an artist orphotographer can easily locate these materials for post-capturerendering. Specifically, automatic differentiation of differentmaterials automatically provides the location or regions which includethe different materials, which can then be accessed by an artist orphotographer for post-capture rendering. Thus, the artist orphotographer has the additional metadata identifying materials in thescene as a resource for rendering the scene.

In step 406, materials for objects in the scene are identified, usingmatches between the spectral profile information for the scene againstthe database. For example, if the coefficients of an object match (orare within a given similarity range as) the coefficients of a curve inthe database, the material corresponding to the matching curve in thedatabase is assigned to the relevant spectral profile information. Thiscan be done, for example, by employing correlation analysis between thespectral profile and the database.

In step 407, metadata which identifies materials for objects in thescene is constructed. Using the metadata, it is possible to determine alocation of one or more objects in the scene comprised of a particularidentified material. The metadata can be constructed as shown in table1202 in FIG. 12. Typical image metadata may include descriptionmetadata, file metadata, origin metadata, camera metadata, advancedphoto metadata and image metadata. Description metadata may contain datasuch as title, subject, rating, tags and comments of the image. Filemetadata may contain data such as file name, item type (e.g. jpeg),folder path, data created, data modified, size and owner name. Originmetadata may contain information such authors, data taken, program name,data acquired and copyright information. Camera metadata could containinformation such as camera maker, camera model, f-stop, exposure time,ISO speed, exposure bias, focal length, maximum aperture, metering mode,subject distance, flash mode, flash energy and 35 mm focal length.Advanced photo metadata could contain information such as lens maker,lens model, camera serial number, contrast, brightness, light source,exposure program, saturation, sharpness, white balance, photometricinterpretation, digital zoom and EXIF version. Image metadata couldinclude data such as image ID, dimensions, width, height, horizontalresolution, vertical resolution, bit depth, compression, resolutionunit, color representation and compressed bits/pixel. Obviously,metadata is not standardized, and it could contain more information suchas GPS coordinates, information of depth map, light field and apertureinformation. One example of material metadata could be composed ofeigenvectors (or an index to a set of eigenvectors) that arepre-calculated and a list of materials with corresponding coordinatesthat encompass the area of the material and the coefficients of theeigenvectors that is the spectral signature of the material. Forexample, in the case of Material 1 in the example of FIG. 12, itcorresponds to the area delimited by coordinates (x1 l,y1 d), (x1 l, y1u), (x1 r,y1 d) and (x1 r,y1 u) and its coefficients are given by thevector A1. In this example, only 4 rectangular coordinates are used fordelimitation, but an arbitrary number of points delimiting the region ofthe material in the image could be used to represent various types ofshapes.

In step 408, the metadata is embedded with the image data for the scene.For example, the metadata can be embedded as additional data for eachpixel in the scene. This method may be useful in a wide assortment ofsituations, as the pixel data can be compressed and offloaded to anapplication (or elsewhere) for processing. Alternatively, the metadatacan be embedded by constructing an array for each respective materialcorresponding to pixels in the image, and indicating pixels of thatmaterial with values in the array. This latter method may be moreefficient in scenes with a relatively small number of materials. In thatregard, the metadata can be constructed as a spatial mask, and thisspatial mask can be used as a metadata that is superimposed over thecaptured RGB image.

In step 409, the image data for the scene is rendered by using themetadata that identifies the material for objects in the scene. In thatregard, image data having similar tri-stimulus values can rendereddifferently in dependence on the metadata. For example, using theexample above, an artist could use the information indicating therespective locations of the hair and skin to adjust shadow detail orother effects for the hair and skin appropriately (and separately). Inone example, management of image data having similar tri-stimulus valuesis directed differently in an output-referred color space in dependenceon the metadata. For example, a photographer could use the locatedmaterials to separate an image into separate layers, which could then beadjusted independently, e.g., in Adobe Photoshop™. In one practicalexample, cosmetics with different spectral signatures can berespectively applied to different people in a scene, and the metadatacan be used to identify a person in the scene using the spectralsignature of a cosmetic applied to that person.

In another example embodiment shown in FIG. 2C, non-volatile memory 56is an example of a non-transitory computer-readable memory medium,having retrievably stored thereon image capture module 370 as describedherein. According to this example embodiment, the image capture module370 includes at least a preview capture module 371 for capturing previewimage data of a scene, a designation module 372 for accepting adesignation of a region of interest in the preview image data, aspectral capture module 373 for capturing spectral image data of thescene, a calculation module 374 for calculating spectral profileinformation for the region of interest by using the captured spectralimage data for the scene, an access module 375 for accessing a databaseof plural spectral profiles of which each profile maps a material to acorresponding spectral profile reflected therefrom, a matching module376 for matching the spectral profile information for the region ofinterest against the database, an identification module 377 foridentifying materials for objects in the region of interest by usingmatches between the spectral profile information for the region ofinterest against the database, a construction module 378 forconstructing metadata which identifies materials for objects in theregion of interest and which identifies location of the region ofinterest relative to the scene, and a storage module 379 for storing themetadata together with image data for the scene. These modules will bediscussed in more detail below with respect to FIG. 3B.

Additionally, as shown in FIG. 2C, non-volatile memory 56 also includesimage data 251, which includes image data from a scene. The image datafor the scene may also be embedded with metadata which identifiesmaterials for objects in the scene. Non-volatile memory 56 furtherstores spectral profile information 252. Spectral profile information252 includes information indicating the spectral signature of objects inthe region of interest, and the respective profile information ismatched against a database of predetermined spectral profiles 253 inorder to identify the materials of the object. Each of these elementswill be described more fully below.

FIG. 3B is a view for explaining an image capture module according tothis example embodiment. As previously discussed with respect to FIG.2C, image capture module 370 comprises computer-executable process stepsstored on a non-transitory computer-readable storage medium, such asnon-volatile memory 56. More or less modules may be used, and otherarchitectures are possible.

As shown in FIG. 3B, image capture module 370 includes at least acapture module 371 which captures preview image data of a scene. To thatend, preview capture module communicates with image sensor 14.Additionally, preview capture module 371 communicates with image displayunit 28, for example to transmit a preview image to be displayed onimage display unit 28 so that a user can select a region of interest.Preview capture module 371 further communicates with designation module372, for example to provide preview image data for designation of aregion of interest.

Designation module 372 accepts a designation of a region of interest inthe preview image data. Thus, designation module 372 is connected topreview capture module 371 to receive the captured preview image data.Designation module 372 is further connected to operation unit 70 toreceive a designation of a region of interest from the user via theoperation unit, such as a user's touch on a touch screen of operationunit 70. Designation module 372 also communicates with spectral capturemodule 373, to provide the designation of a region of interest forfurther processing.

Spectral capture module 373 captures spectral image data of the scene.To that end, spectral capture module 373 is connected to imaging system150, and/or may be connected to image sensor 14. Thus, spectral capturemodule 373 may communicate with different hardware depending on how thespectral data is obtained (e.g., from image sensor 14 if image sensor 14can capture such data alone, or from imaging system 150 if image sensor14 is a conventional RGB sensor). Spectral capture module 373 furtherprovides captured spectral image data to calculation module 374.

Calculation module 374 calculates spectral profile information for theregion of interest by using the captured spectral image data for thescene. The calculated spectral profile information may be stored, forexample, as spectral profile information 252 in non-volatile memory 56,as shown in FIG. 2C.

Access module 375 accesses a database of plural spectral profiles, forexample database 253 stored in non-volatile memory 56, as shown in FIG.2C. Each profile maps a material to a corresponding spectral profilereflected therefrom. Access module 375 further communicates withmatching module 376, which matches the spectral profile information forthe region of interest against the database. Identification module 377identifies materials for objects in the region of interest by usingmatches between the spectral profile information for the region ofinterest against the database.

Construction module 378 constructs metadata which identifies materialsfor objects in the region of interest and which identifies location ofthe region of interest relative to the scene.

Storage module 379 stores the metadata together with image data for thescene, for example in non-volatile memory 56. In one example, storagemodule 379 embeds the metadata with the image data for the scene. Theresultant embedded image data may be stored with other image data, forexample as image data 251 in non-volatile memory 56 shown in FIG. 2C.

FIG. 4B is a flow diagram for explaining processing in the image capturedevice shown in FIG. 1 according to the example embodiment.

Briefly, as shown in FIG. 4B, preview image data of a scene is captured.A designation of a region of interest is accepted in the preview imagedata. Spectral image data of the scene is captured, and spectral profileinformation for the region of interest is calculated by using thecaptured spectral image data for the scene. A database of pluralspectral profiles is accessed, of which each profile maps a material toa corresponding spectral profile reflected therefrom. The spectralprofile information for the region of interest is matched against thedatabase, and materials for objects in the region of interest areidentified by using matches between the spectral profile information forthe region of interest against the database. Metadata which identifiesmaterials for objects in the region of interest and which identifieslocation of the region of interest relative to the scene is constructed.The metadata is stored together with image data for the scene.

In more detail, in step 451, a capture of preview image data of a sceneis instructed. In that regard, the preview image data capture could beinstructed by a user, or automatically by the image capture apparatus.For example, image capture apparatus 100 could be constructed toautomatically capture preview image data in certain modes, such as amaterial identification mode for capturing spectral information as wellas image data. In another example, image capture apparatus 100 could beconstructed to automatically capture preview image data as a defaultsetting.

In step 452, preview image data of the scene is captured. For example,image data of the scene currently sensed by image sensor 14 is capturedand displayed on image display unit 28, thereby effecting a display of apreview image. In that regard, in order to “set” or hold a current sceneas a preview image, image capture device 100 might provide a pause ofthe current image in response to, for example, a half-stroke of shutterbutton 310, or a tap on a touchscreen, such as that of image displayunit 28.

In step 453, spectral image data of the scene is captured. Inparticular, spectral information is captured along with raw image databy image sensor 14 (if image sensor 14 is capable of capturingsufficient spectral data on its own), or by a combination of imagesensor 14 and imaging system 150 (if image sensor 14 is not capable ofcapturing sufficient spectral data on its own). Embodiments forcapturing the spectral information are described above with respect toFIGS. 1C to 1G. If image sensor 14 is capable of capturing sufficientspectral data, stored image data (e.g., image data 251) can be comprisedof the captured spectral image data. In that regard, spectral data maybe captured for the entire scene, even though spectral profileinformation may ultimately be calculated only for a region of interest.

In some instances, the captured spectral image data may below-resolution image data having three (3) or less components, e.g.,only a few channels such as RGB, or even, in some cases, merely blackand white. On the other hand, the spectral information may also includea high number of spectral components including, for example, five ormore channels of color information, including a red-like channel, agreen-yellow-like channel, a green-like channel, a blue-green-likechannel, and a blue-like channel. If image sensor 14 captures thespectral image data, the stored image data can be comprised oftri-stimulus device independent image data, e.g., XYZ image data derivedfrom the captured spectral image data.

In step 454, a region of interest is designated in the captured previewimage data.

In that regard, FIG. 5A is a view for explaining designation of a regionof interest with the image capture device shown in FIG. 1. Inparticular, a rear view of image capture apparatus 100 having imagedisplay unit 28 is provided in FIG. 5A. According to this exampleembodiment, a user interface which includes a preview image based oncaptured image data of a scene is displayed on the image display unit28.

The user controlling the image capture device 100 views the previewimage displayed on the image display unit 28 as shown in FIG. 5A, anddecides a region of the scene in which to differentiate locationscomprised of different materials.

In particular, an artist or photographer may only want to differentiatebetween materials in certain regions of a scene. For example, while anartist or photographer may be concerned with differentiating betweensimilarly-colored areas in the foreground of a scene (e.g., making adistinction between a black velvet jacket over black leather pants), theartist may nonetheless be unconcerned about differentiating betweenobjects or regions comprising the background of the scene. In such acase, calculating spectral profile information and identifying ordifferentiating between materials for the entire scene could wastememory space and processing resources. Accordingly, the presentembodiment allows the user to narrow down the scene to one or moreregions of interest for which to calculate spectral profile information.

FIG. 5B is a view for explaining acceptance of a designation of a regionof interest according to one example embodiment. As shown in FIG. 5B,the preview image displayed on the image display unit 28 depicts animage divided into a plurality of regions.

Multiple different methods of segmenting the image into regions arepossible. In one example, RGB (or other color scheme) values aredetermined for each pixel in the preview image, and pixels havingsubstantially the same RGB values (or within a certain range ortolerance) are determined to be included in the same ROI. Alternatively,the ROI can be actively determined. For example, when the userdesignates the ROI in the preview image, the image capture device candetermine which pixels of the image which are included in the ROI. Forexample, a spatial filtering algorithm is executed to determine theedges of the ROI. Thus, the user “grabs” a region. Of course, any othersuitable processes for dividing the image into regions can also be used.Additionally, the user may adjust the size of the regions relative tothe image displayed.

In FIG. 5B, the preview image includes three regions. In one region ofthe preview image, a table and lamp are displayed. In another region, aperson is displayed. In a third region, a floor area is displayed. Asshown in FIG. 2B, the user designates the region including the person.

Thus, as shown in FIG. 5C, the region including the person is displayed,along with identification of different materials comprising the person.The different materials of the region of interest can be identified anddata of the identified materials can be embedded with metadata for thescene, as discussed more fully below.

The user interfaces depicted in FIGS. 5A to 5C are merely examples ofuser interfaces which can be displayed by the user interface accordingto this example embodiment. It should be understood that other types ofsuitable interfaces can also be displayed. In addition, other selectionmethods of a region of interest may be used, e.g., tapping with twofingers, a zoom method, voice commands, gaze tracking, and so on.

In step 455, spectral profile information for the region of interest iscalculated by using the captured spectral image data for the scene. Asmentioned above, spectral profile information may be obtained from magesensor 14 (if capable of capturing sufficient spectral data on its own)or from a combination of image sensor 14 and imaging system 150 (ifimage sensor 14 is not capable of capturing sufficient spectral data onits own).

Spectral data gathered by imaging system 150 (or image sensor 14, ifacting alone) is converted into a spectral reflectance curve, generallyin the range from 400 to 700 nm of visible light. For example, in anexample embodiment in which each pixel has five channels, each pixel isintegrated to produce five digital signals, one signal for each channel.Each channel is tuned to a spectral band within the visible spectrum.Therefore, the digital signal for each channel corresponds to arespective spectral reflectance curve within the visible spectrum.

In that regard, spectral data may have up to 61 or more separate values.Comparing all of these values can be relatively inefficient.Accordingly, since spectral reflectance curves are generally smooth, itis ordinarily possible to use less values (i.e., less than the 61discrete values), and eigenvectors can be used to reduce the requiredprocessing.

By assuming the relative smoothness of most of spectral reflectancecurves it is possible to reduce the number of components of spectraldata to six eigenvectors by performing eigenvector analysis. Atransformation from the six capture signals to the coefficients ofeigenvectors can be produced by a training set of captured images ofobjects with known representative spectral reflectances. Once the imageis captured, the transformation is used to calculate the coefficients ofthe eigenvectors for each pixel of the image.

Specifically, eigenvectors and their coefficients represent the spectraldata. The pre-calculated eigenvectors are used to decompose the capturedspectral curves into coefficients, which can then be compared withcoefficients in the database. The pre-calculated eigenvectors can begenerated before image capture from common captured spectralreflectances, such as skin, clothes, hair and the like. Alternatively,eigenvectors could be pre-calculated for every possible reflectance,although this approach might require significant resources.

In one approach, the spectral reflectance of a collection of objectsR_(λ) _(—) _(collection□) is statistically analyzed. Eigenvectoranalysis is performed and 6 eigenvectors e_(i) (where i=1 to 6) arepre-calculated. Any reflectance R_(λ) _(—) _(j) (where j=1 to m, where mis the number of objects in the collection) in the collection of objectscould be reconstructed by combining the eigenvectors e_(j).

Meanwhile, the estimation of the spectral reflectance for a capturedobject j is given by R_(λ) _(—) _(j) _(—) _(estimation)=Σa_(i)*e_(i)where a_(i) are the coefficients of the eigenvectors for object j. Thecoefficients of the eigenvectors (represented here by a vector A_(j)whose dimensions are i by 1) can be estimated from captured digitalsignals D_(i) of object j by a pre-calculated transformation T fromcaptured digital signals to eigenvectors: Aj=T*D_(j). Accordingly, it ispossible to obtain the coefficients of the eigenvectors from thecaptured spectral reflectance curves, which can then be compared withcoefficients of eigenvectors from the database of plural spectralprofiles to see if there is a match.

In some example embodiments such as that shown in FIG. 1F, due tohigh-number of components of spectral information, it is difficult todeal with spectral data as signatures for objects. One possibility todeal with this burden is by relating coefficients of eigenvectors Ajassociated to a particular object j.

In such a configuration, the measured spectra can be decomposed by thepre-calculated eigenvectors e_(i) as follows: Aj=R_(λ) _(—)_(j)*pinv(e_(i)), where pinv is the pseudo-inverse operation.

A concrete example of calculating spectral profile information from thecaptured image data for the scene will briefly be described with respectto FIGS. 6 to 11.

In this example, assume a model of African origin whose face skin has aspectral reflectance R_skin and who has black hair with spectralreflectance R_hair. The typical spectral reflectance curves are shown inFIG. 6. It is clear from FIG. 6 that hair and skin have very distinctspectral reflectance properties.

First, assume the model is imaged under typical photographic studiohalogen lamps (whose spectral power distribution is shown in FIG. 7) andthe model pictures are taken by a conventional professional digital SLRwhose typical red-green-blue spectral sensitivities are shown in FIG. 8.When the digital images are captured, they include average values ofRed_hair=24, Green_hair=14 and Blue_hair=7 for hair and average valuesof Red_skin=24, Green_skin=11 and Blue_skin=5 for skin. The cameravalues for dark skin and black hair are extremely similar, making themsomewhat undistinguishable.

On the other hand, an imaging system that has a secondary spectralmeasurement sensor (e.g., any of FIGS. 1C to 1G) or an image sensor 14with high spectral resolution captures spectral reflectance values formultiple regions of the image including hair and skin, respectivelyR_hair and R_skin. These measurements correspond to what is depicted inFIG. 6.

When the coefficient of eigenvectors are calculated for the capturedblack hair data it gives the following values are produced:A_hair=[0.006, −0.011, −0.001, −0.007, 0.017, 0.118], while the valuesfor dark skin are given by A_skin=[0.0002, −0.029, −0.027, −0.035,−0.043, 0.429]. In this case, the spectral signatures given by thecoefficients of eigenvectors are distinct between dark skin and blackhair. These eigenvectors are compared with a database of plural spectralprofiles such as database of spectral profiles 253 to identify materialsfor objects in the region of interest, as described more fully below.Additional details of the above processes can be found in U.S.application Ser. No. 13/871,826, filed Feb. 24, 2011, titled “ImageCapture And Post-Capture Processing”, by John Haikin, et. al, thecontents of which are incorporated herein by reference.

Returning to FIG. 4B, in step 456, a database of plural spectralprofiles is accessed. The database of plural spectral profiles may bestored in non-volatile memory 56, as shown by database of spectralprofiles 253 in FIG. 2C. In another embodiment, the database of pluralspectral profiles could be stored remotely in a server, provided thatsuch server can be accessed from image capture apparatus 100, i.e., aslong as image capture apparatus 100 has remote data access capabilities.Each of the plural spectral profiles maps a material to a correspondingspectral profile reflected therefrom.

FIG. 9 depicts an example of such a database. More specifically, FIG. 9depicts a spectral database (such as the Vrhel database: Vrhel, M. J.,R. Gershon, and L. S. Iwan, Measurement and analysis of objectreflectance spectra, Color Res. and Appl., 19, 4-9, 1994, the contentsof which are incorporated by reference herein. This database iscomprised by spectral measurement of 170 objects. In that regard, forpurposes of conciseness, the full database is not shown in FIG. 9. Thedatabase is one example of a pre-loaded set of spectral profiles in theform of computed eigenvectors and a look-up table (LUT) with typicalspectral signatures (coefficients of eigenvectors) of most commonlyimaged objects, such as skin, hair, vegetation, sky, etc.

Eigenvector analysis is performed for this collection of spectralreflectances, and the first 6 eigenvectors are shown in FIG. 10.

In step 457, the spectral profile information for the scene is matchedagainst the database.

In particular, the coefficients of eigenvectors calculated in step 455for the captured black hair data A_hair=[0.006, −0.011, −0.001, −0.007,0.017, 0.118] and the dark skin data A_skin=[0.0002, −0.029, −0.027,−0.035, −0.043, 0.429] are compared with the plural profiles of spectralsignatures accessed in step 404 to see if there are matches withspectral signatures of pre-identified objects in the database. If thereare matches, the respective spectral signatures are then used to segmentareas of the region of interest with different spectral properties.

In that regard, the spectral profiles may be spectral profiles having arelatively low number of spectral components. For example, the spectralprofiles can also be low-resolution spectral profiles having three (3)or less components. In particular, it may be unnecessary and impracticalto attempt to specifically identify the exact material for each objectin the region of interest. For example, outside of a specific setting inwhich all potential materials are known, it may not be possible tospecifically identify an exact material, as this would require anenormous database of plural spectral profiles for all possiblematerials.

Nevertheless, spectral profiles with a relatively low number of spectralcomponents still can be used to differentiate between distinct areasmade up of different materials, so that an artist or photographer caneasily locate these areas for post-capture rendering. Thus, the artistor photographer has the additional metadata identifying locations ofdifferent materials in the scene as a resource for rendering the scene.

In step 458, materials for objects in the region of interest areidentified, using matches between the spectral profile information forthe region of interest against the database. For example, if thecoefficients of an object match (or are within a given similarity rangeas) the coefficients of a curve in the database, the materialcorresponding to the matching curve in the database is assigned to therelevant spectral profile information.

In step 459, metadata which identifies materials for objects in theregion of interest is constructed. Using the metadata, it is possible todetermine a location of one or more objects in the region of interestcomprised of a particular identified material. The metadata may alsoidentify the location of the region of interest relative to the rest ofthe scene. After the image is captured, the material metadata can beread by a photo edition program and an image layer that superimposes theoriginal image can be created that contains the spectral signatures foreach material. This layer can be used to automatically segment eachregion with different material making easy for the user to distinguishregions of the image with different materials and do appropriate editionand rendering for each segmented region. Another application is materialedition, a process in which the user of a photo edition program caneasily identify and segment areas of the image and change the material.This metadata also enables other applications such as identification ofmaterials in the scene by comparing the spectral signature to a databaseof spectral signatures.

In step 460, the constructed metadata is stored. For example, themetadata can be embedded with the image data for the scene. In thatregard, the metadata can be embedded as additional data for each pixelin the region of interest. This method may be useful in a wideassortment of situations, as the pixel data can be compressed andoffloaded to an application (or elsewhere) for processing.Alternatively, the metadata can be embedded by constructing an array foreach respective material corresponding to pixels in the region ofinterest, and indicating pixels of that material with values in thearray. This latter method may be more efficient in scenes with arelatively small number of materials. In one example, the metadata canbe constructed as a spatial mask, and this spatial mask can be used as ametadata that is superimposed over the captured RGB image of the regionof interest.

In step 461, there is a determination of whether another region ofinterest is to be selected by the user. For example, the photographermay notice another region of the scene for which the photographer wishesto differentiate between materials. If the user wishes to select anotherregion of interest, the process proceeds to step 454 to designateanother region of interest.

In one embodiment, in a case that the metadata identifies multiplesub-regions comprised of different materials in the designated region ofinterest, the sub-regions are made available for designation as separateregions of interest in a subsequent designation of a region of interest.Thus, iteration of the above processes can provide betterdifferentiation even at the point of selecting a ROI.

If another region of interest is not to be selected, the processproceeds to step 462.

In step 462, the stored image data for the region of interest isrendered by using the metadata that identifies the material for objectsin the region of interest. Thus, image data having similar tri-stimulusvalues can rendered differently in dependence on the metadata. Forexample, using the example above, an artist could use the informationindicating the respective locations of the hair and skin to adjustshadow detail or other effects for the hair and skin appropriately (andseparately). In one example, management of image data having similartri-stimulus values is directed differently in an output-referred colorspace in dependence on the metadata. For example, a photographer coulduse the located materials to separate the region of interest intoseparate layers, which could then be adjusted independently, e.g., inAdobe Photoshop™. In one practical example, cosmetics with differentspectral signatures can be respectively applied to different people in aregion of interest, and the metadata can be used to identify a person inthe region of interest using the spectral signature of a cosmeticapplied to that person.

FIG. 11 is a view for explaining the use of spectral reflectances toidentify distinct areas in a captured image.

In particular, FIG. 11 depicts different spectral reflectance curves forskin and hair of two separate subjects. As can be seen from FIG. 11, therespective skin and hair of subjects A and B clearly have differentspectral reflectances. Thus, according to the arrangements describedabove, the location of one or more objects or regions in the scenecomprised of these materials can be distinctly identified.

FIGS. 12 and 13 are views for explaining metadata according to exampleembodiments.

In particular, FIG. 12 shows a table 1202 which describes differenttypes of metadata, as discussed above with respect to FIG. 4A.Meanwhile, as shown in FIG. 13, a rectangle 1301 may be shown by usingthe coordinates information shown in FIG. 12. For example, the textinformation 1300 (“apple”) may be shown by using a coefficienteigenvector from the metadata as shown in FIG. 12. In one example,information regarding coefficient eigenvectors and corresponding textinformation (e.g., “apple”) can be stored in a table in a memory.

According to other embodiments contemplated by the present disclosure,example embodiments may include a computer processor such as a singlecore or multi-core central processing unit (CPU) or micro-processingunit (MPU), which is constructed to realize the functionality describedabove. The computer processor might be incorporated in a stand-aloneapparatus or in a multi-component apparatus, or might comprise multiplecomputer processors which are constructed to work together to realizesuch functionality. The computer processor or processors execute acomputer-executable program (sometimes referred to ascomputer-executable instructions or computer-executable code) to performsome or all of the above-described functions. The computer-executableprogram may be pre-stored in the computer processor(s), or the computerprocessor(s) may be functionally connected for access to anon-transitory computer-readable storage medium on which thecomputer-executable program or program steps are stored. For thesepurposes, access to the non-transitory computer-readable storage mediummay be a local access such as by access via a local memory busstructure, or may be a remote access such as by access via a wired orwireless network or Internet. The computer processor(s) may thereafterbe operated to execute the computer-executable program or program stepsto perform functions of the above-described embodiments.

According to still further embodiments contemplated by the presentdisclosure, example embodiments may include methods in which thefunctionality described above is performed by a computer processor suchas a single core or multi-core central processing unit (CPU) ormicro-processing unit (MPU). As explained above, the computer processormight be incorporated in a stand-alone apparatus or in a multi-componentapparatus, or might comprise multiple computer processors which worktogether to perform such functionality. The computer processor orprocessors execute a computer-executable program (sometimes referred toas computer-executable instructions or computer-executable code) toperform some or all of the above-described functions. Thecomputer-executable program may be pre-stored in the computerprocessor(s), or the computer processor(s) may be functionally connectedfor access to a non-transitory computer-readable storage medium on whichthe computer-executable program or program steps are stored. Access tothe non-transitory computer-readable storage medium may form part of themethod of the embodiment. For these purposes, access to thenon-transitory computer-readable storage medium may be a local accesssuch as by access via a local memory bus structure, or may be a remoteaccess such as by access via a wired or wireless network or Internet.The computer processor(s) is/are thereafter operated to execute thecomputer-executable program or program steps to perform functions of theabove-described embodiments.

The non-transitory computer-readable storage medium on which acomputer-executable program or program steps are stored may be any of awide variety of tangible storage devices which are constructed toretrievably store data, including, for example, any of a flexible disk(floppy disk), a hard disk, an optical disk, a magneto-optical disk, acompact disc (CD), a digital versatile disc (DVD), micro-drive, a readonly memory (ROM), random access memory (RAM), erasable programmableread only memory (EPROM), electrically erasable programmable read onlymemory (EEPROM), dynamic random access memory (DRAM), video RAM (VRAM),a magnetic tape or card, optical card, nanosystem, molecular memoryintegrated circuit, redundant array of independent disks (RAID), anonvolatile memory card, a flash memory device, a storage of distributedcomputing systems and the like. The storage medium may be a functionexpansion unit removably inserted in and/or remotely accessed by theapparatus or system for use with the computer processor(s).

By matching spectral profiles of objects in a scene so as to identifymaterials for the objects, and storing the materials in metadatatogether with image data for the scene for use during post-capturerendering, it is ordinarily possible to automatically identify distinctareas of an image for separate post-processing, without requiring theintervention of an artist or photographer.

This disclosure has provided a detailed description with respect toparticular representative embodiments. It is understood that the scopeof the appended claims is not limited to the above-described embodimentsand that various changes and modifications may be made without departingfrom the scope of the claims.

1. An image capture method comprising: capturing image data of a scene;obtaining spectral profile information for the scene; accessing adatabase of plural spectral profiles each of which maps a material to acorresponding spectral profile reflected therefrom; matching thespectral profile information for the scene against the database;identifying materials for objects in the scene by using matches betweenthe spectral profile information for the scene against the database;constructing metadata which identifies materials for objects in thescene; and embedding the metadata with the image data for the scene. 2.The method according to claim 1, wherein the spectral profiles arecomprised of a low number of spectral components.
 3. The methodaccording to claim 1, further comprising rendering of the image data forthe scene by using the metadata that identifies the material for objectsin the scene.
 4. The method according to claim 1, wherein the spectralprofile information for the scene is calculated from the captured imagedata of the scene.
 5. The method according to claim 1, wherein the imagedata is comprised of tri-stimulus device independent image data.
 6. Themethod according to claim 5, further comprising rendering of the imagedata for the scene by using the metadata that identifies the materialfor objects in the scene, and wherein image data having similartri-stimulus values is rendered differently in dependence on themetadata.
 7. The method according to claim 6, wherein management ofimage data having similar tri-stimulus values is directed differently inan output-referred color space in dependence on the metadata.
 8. Themethod according to claim 1, wherein the metadata is embedded asadditional data for each pixel in the scene.
 9. The method according toclaim 1, wherein the metadata is embedded by constructing an arraycorresponding to pixels in the image for each respective material, andindicating pixels of that material with values in the array.
 10. Themethod according to claim 1, wherein cosmetics with different spectralsignatures are respectively applied to different people in the scene,and wherein the metadata is used to identify a person in the scene usingthe spectral signature of a cosmetic applied to that person.
 11. Themethod according to claim 1, further comprising determining a locationof one or more objects in the scene comprised of a particular identifiedmaterial.
 12. An image capture apparatus, comprising: acomputer-readable memory constructed to store computer-executableprocess steps; and a processor constructed to execute thecomputer-executable process steps stored in the memory; wherein theprocess steps stored in the memory cause the processor to: capture imagedata of a scene; obtain spectral profile information for the scene;access a database of plural spectral profiles each of which maps amaterial to a corresponding spectral profile reflected therefrom; matchthe spectral profile information for the scene against the database;identify materials for objects in the scene by using matches between thespectral profile information for the scene against the database;construct metadata which identifies materials for objects in the scene;and embed the metadata with the image data for the scene.
 13. Theapparatus according to claim 12, wherein the spectral profiles arecomprised of a low number of spectral components.
 14. The apparatusaccording to claim 12, further comprising a step of rendering of theimage data for the scene by using the metadata that identifies thematerial for objects in the scene.
 15. The apparatus according to claim12, wherein the spectral profile information for the scene is calculatedfrom the captured image data of the scene.
 16. The apparatus accordingto claim 12, wherein the image data is comprised of tri-stimulus deviceindependent image data.
 17. The apparatus according to claim 16, furthercomprising rendering of the image data for the scene by using themetadata that identifies the material for objects in the scene, andwherein image data having similar tri-stimulus values is rendereddifferently in dependence on the metadata.
 18. The apparatus accordingto claim 17, wherein management of image data having similartri-stimulus values is directed differently in an output-referred colorspace in dependence on the metadata.
 19. The apparatus according toclaim 12, wherein the metadata is embedded as additional data for eachpixel in the scene.
 20. The apparatus according to claim 12, wherein themetadata is embedded by constructing an array corresponding to pixels inthe image for each respective material, and indicating pixels of thatmaterial with values in the array.
 21. The apparatus according to claim12, wherein cosmetics with different spectral signatures arerespectively applied to different people in the scene, and wherein themetadata is used to identify a person in the scene using the spectralsignature of a cosmetic applied to that person.
 22. The apparatusaccording to claim 12, wherein the process steps further cause thecomputer to determine a location of one or more objects in the scenecomprised of a particular identified material.
 23. An image capturemodule comprising: a capture module for capturing image data of a scene;a obtaining module for obtaining spectral profile information for thescene; an access module for accessing a database of plural spectralprofiles each of which maps a material to a corresponding spectralprofile reflected therefrom; a matching module for matching the spectralprofile information for the scene against the database; anidentification module for identifying materials for objects in the sceneby using matches between the spectral profile information for the sceneagainst the database; a construction module for constructing metadatawhich identifies materials for objects in the scene; and an embeddingmodule for embedding the metadata with the image data for the scene. 24.The module according to claim 23, wherein the spectral profiles arecomprised of a low number of spectral components.
 25. The moduleaccording to claim 23, further comprising rendering of the image datafor the scene by using the metadata that identifies the material forobjects in the scene.
 26. The module according to claim 23, wherein thespectral profile information for the scene is calculated from thecaptured image data of the scene.
 27. The module according to claim 23,wherein the image data is comprised of tri-stimulus device independentimage data.
 28. The module according to claim 27, further comprisingrendering of the image data for the scene by using the metadata thatidentifies the material for objects in the scene, and wherein image datahaving similar tri-stimulus values is rendered differently in dependenceon the metadata.
 29. The module according to claim 28, whereinmanagement of image data having similar tri-stimulus values is directeddifferently in an output-referred color space in dependence on themetadata.
 30. The module according to claim 23, wherein the metadata isembedded as additional data for each pixel in the scene.
 31. The moduleaccording to claim 23, wherein the metadata is embedded by constructingan array corresponding to pixels in the image for each respectivematerial, and indicating pixels of that material with values in thearray.
 32. The module according to claim 23, wherein cosmetics withdifferent spectral signatures are respectively applied to differentpeople in the scene, and wherein the metadata is used to identify aperson in the scene using the spectral signature of a cosmetic appliedto that person.
 33. The module according to claim 23, further comprisinga location determination module for determining a location of one ormore objects in the scene comprised of a particular identified material.34. A computer-readable storage medium retrievably storingcomputer-executable process steps for causing a computer to perform animage capture method, the method comprising: capturing image data of ascene; obtaining spectral profile information for the scene; accessing adatabase of plural spectral profiles each of which maps a material to acorresponding spectral profile reflected therefrom; matching thespectral profile information for the scene against the database;identifying materials for objects in the scene by using matches betweenthe spectral profile information for the scene against the database;constructing metadata which identifies materials for objects in thescene; and embedding the metadata with the image data for the scene. 35.The computer-readable storage medium according to claim 34, wherein thespectral profiles are comprised of a low number of spectral components.36. The computer-readable storage medium according to claim 34, furthercomprising rendering of the image data for the scene by using themetadata that identifies the material for objects in the scene.
 37. Thecomputer-readable storage medium according to claim 34, wherein thespectral profile information for the scene is calculated from thecaptured image data of the scene.
 38. The computer-readable storagemedium according to claim 34, wherein the image data is comprised oftri-stimulus device independent image data.
 39. The computer-readablestorage medium according to claim 38, further comprising rendering ofthe image data for the scene by using the metadata that identifies thematerial for objects in the scene, and wherein image data having similartri-stimulus values is rendered differently in dependence on themetadata.
 40. The computer-readable storage medium according to claim39, wherein management of image data having similar tri-stimulus valuesis directed differently in an output-referred color space in dependenceon the metadata.
 41. The computer-readable storage medium according toclaim 34, wherein the metadata is embedded as additional data for eachpixel in the scene.
 42. The computer-readable storage medium accordingto claim 34, wherein the metadata is embedded by constructing an arraycorresponding to pixels in the image for each respective material, andindicating pixels of that material with values in the array.
 43. Thecomputer-readable storage medium according to claim 34, whereincosmetics with different spectral signatures are respectively applied todifferent people in the scene, and wherein the metadata is used toidentify a person in the scene using the spectral signature of acosmetic applied to that person.
 44. The computer-readable storagemedium according to claim 34, wherein the method further comprisesdetermining a location of one or more objects in the scene comprised ofa particular identified material.
 45. An image capture methodcomprising: capturing preview image data of a scene; accepting adesignation of a region of interest in the preview image data; capturingspectral image data of the scene; calculating spectral profileinformation for the region of interest by using the captured spectralimage data for the scene; accessing a database of plural spectralprofiles of which each profile maps a material to a correspondingspectral profile reflected therefrom; matching the spectral profileinformation for the region of interest against the database; identifyingmaterials for objects in the region of interest by using matches betweenthe spectral profile information for the region of interest against thedatabase; constructing metadata which identifies materials for objectsin the region of interest and which identifies location of the region ofinterest relative to the scene; and storing the metadata together withimage data for the scene.
 46. The method according to claim 45, whereinthe spectral profiles are low-resolution spectral profiles having three(3) or less components.
 47. The method according to claim 45, furthercomprising rendering of the image data for the region of interest byusing the metadata that identifies the material for objects in theregion of interest.
 48. The method according to claim 45, wherein thestored image data is comprised of the captured spectral image data. 49.The method according to claim 45, wherein the stored image data iscomprised of tri-stimulus device independent image data derived from thecaptured spectral image data.
 50. The method according to claim 49,further comprising rendering of the stored image data for the region ofinterest by using the metadata that identifies the material for objectsin the region of interest, and wherein image data having similartri-stimulus values is rendered differently in dependence on themetadata.
 51. The method according to claim 45, wherein in a case thatthe metadata identifies multiple sub-regions comprised of differentmaterials in the designated region of interest, the sub-regions are madeavailable for designation as separate regions of interest in asubsequent designation of a region of interest.
 52. An image captureapparatus, comprising: a computer-readable memory constructed to storecomputer-executable process steps; and a processor constructed toexecute the computer-executable process steps stored in the memory;wherein the process steps stored in the memory cause the processor to:capture preview image data of a scene; accept a designation of a regionof interest in the preview image data; capture spectral image data ofthe scene; calculate spectral profile information for the region ofinterest by using the captured spectral image data for the scene; accessa database of plural spectral profiles of which each profile maps amaterial to a corresponding spectral profile reflected therefrom; matchthe spectral profile information for the region of interest against thedatabase; identify materials for objects in the region of interest byusing matches between the spectral profile information for the region ofinterest against the database; construct metadata which identifiesmaterials for objects in the region of interest and which identifieslocation of the region of interest relative to the scene; and store themetadata together with image data for the scene.
 53. The apparatusaccording to claim 52, wherein the spectral profiles are low-resolutionspectral profiles having three (3) or less components.
 54. The apparatusaccording to claim 52, wherein the process steps further cause theprocessor to render the image data for the region of interest by usingthe metadata that identifies the material for objects in the region ofinterest.
 55. The apparatus according to claim 52, wherein the storedimage data is comprised of the captured spectral image data.
 56. Theapparatus according to claim 52, wherein the stored image data iscomprised of tri-stimulus device independent image data derived from thecaptured spectral image data.
 57. The apparatus according to claim 56,wherein the process steps further cause the processor to render thestored image data for the region of interest by using the metadata thatidentifies the material for objects in the region of interest, andwherein image data having similar tri-stimulus values is rendereddifferently in dependence on the metadata.
 58. The apparatus accordingto claim 52, wherein in a case that the metadata identifies multiplesub-regions comprised of different materials in the designated region ofinterest, the sub-regions are made available for designation as separateregions of interest in a subsequent designation of a region of interest.59. An image capture module comprising: a preview capture module forcapturing preview image data of a scene; a designation module foraccepting a designation of a region of interest in the preview imagedata; a spectral capture module for capturing spectral image data of thescene; a calculation module for calculating spectral profile informationfor the region of interest by using the captured spectral image data forthe scene; an access module for accessing a database of plural spectralprofiles of which each profile maps a material to a correspondingspectral profile reflected therefrom; a matching module for matching thespectral profile information for the region of interest against thedatabase; an identification module for identifying materials for objectsin the region of interest by using matches between the spectral profileinformation for the region of interest against the database; aconstruction module for constructing metadata which identifies materialsfor objects in the region of interest and which identifies location ofthe region of interest relative to the scene; and a storage module forstoring the metadata together with image data for the scene.
 60. Theimage capture module according to claim 59, wherein the spectralprofiles are low-resolution spectral profiles having three (3) or lesscomponents.
 61. The image capture module according to claim 59, whereinthe image data for the region of interest is rendered by using themetadata that identifies the material for objects in the region ofinterest.
 62. The image capture module according to claim 59, whereinthe stored image data is comprised of the captured spectral image data.63. The image capture module according to claim 59, wherein the storedimage data is comprised of tri-stimulus device independent image dataderived from the captured spectral image data.
 64. The image capturemodule according to claim 63, wherein the stored image data for theregion of interest is rendered by using the metadata that identifies thematerial for objects in the region of interest, and wherein image datahaving similar tri-stimulus values is rendered differently in dependenceon the metadata.
 65. The image capture module according to claim 59,wherein in a case that the metadata identifies multiple sub-regionscomprised of different materials in the designated region of interest,the sub-regions are made available for designation as separate regionsof interest in a subsequent designation of a region of interest.
 66. Acomputer-readable storage medium retrievably storing computer-executableprocess steps for causing a computer to perform an image capture method,the method comprising: capturing preview image data of a scene;accepting a designation of a region of interest in the preview imagedata; capturing spectral image data of the scene; calculating spectralprofile information for the region of interest by using the capturedspectral image data for the scene; accessing a database of pluralspectral profiles of which each profile maps a material to acorresponding spectral profile reflected therefrom; matching thespectral profile information for the region of interest against thedatabase; identifying materials for objects in the region of interest byusing matches between the spectral profile information for the region ofinterest against the database; constructing metadata which identifiesmaterials for objects in the region of interest and which identifieslocation of the region of interest relative to the scene; and storingthe metadata together with image data for the scene.
 67. Thecomputer-readable storage medium according to claim 66, wherein thespectral profiles are low-resolution spectral profiles having three (3)or less components.
 68. The computer-readable storage medium accordingto claim 66, wherein the method further comprises rendering of the imagedata for the region of interest by using the metadata that identifiesthe material for objects in the region of interest.
 69. Thecomputer-readable storage medium according to claim 66, wherein thestored image data is comprised of the captured spectral image data. 70.The computer-readable storage medium according to claim 66, wherein thestored image data is comprised of tri-stimulus device independent imagedata derived from the captured spectral image data.
 71. Thecomputer-readable storage medium according to claim 70, wherein themethod further comprises rendering of the stored image data for theregion of interest by using the metadata that identifies the materialfor objects in the region of interest, and wherein image data havingsimilar tri-stimulus values is rendered differently in dependence on themetadata.
 72. The computer-readable storage medium according to claim66, wherein in a case that the metadata identifies multiple sub-regionscomprised of different materials in the designated region of interest,the sub-regions are made available for designation as separate regionsof interest in a subsequent designation of a region of interest.